Fix ORM vs migration files inconsistencies #44221
Open
+141
−53
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:db-migrations
ephraimbuddy
force-pushed
the
investigate-migrations
branch
5 times, most recently
from
031bfd8
to
4ac20ad
Compare
ephraimbuddy
commented
Nov 22, 2024
| local_cols=["asset_id"], | ||
| remote_cols=["id"], | ||
| ondelete="CASCADE", | ||
| ) |
There was a problem hiding this comment.
I moved this to a new file because, for some reason, SQLite could not create this FK in this file. I suspect it is because the asset table was renamed in this migration
There was a problem hiding this comment.
Yeah last I had tried I ran into:
Performing upgrade to the metadata database sqlite:////root/airflow/sqlite/airflow.db
[2024-10-27T22:43:35.430+0000] {migration.py:215} INFO - Context impl SQLiteImpl.
[2024-10-27T22:43:35.431+0000] {migration.py:218} INFO - Will assume non-transactional DDL.
[2024-10-27T22:43:35.432+0000] {migration.py:215} INFO - Context impl SQLiteImpl.
[2024-10-27T22:43:35.433+0000] {migration.py:218} INFO - Will assume non-transactional DDL.
[2024-10-27T22:43:35.433+0000] {db.py:1171} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade 22ed7efa9da2 -> 044f740568ec, Drop ab_user.id foreign key.
INFO [alembic.runtime.migration] Running upgrade 044f740568ec -> d0f1c55954fa, Remove SubDAGs: `is_subdag` & `root_dag_id` columns from DAG table.
INFO [alembic.runtime.migration] Running upgrade d0f1c55954fa -> 0bfc26bc256e, Rename DagModel schedule_interval to timetable_summary.
INFO [alembic.runtime.migration] Running upgrade 0bfc26bc256e -> a2c32e6c7729, Add triggered_by field to DagRun.
INFO [alembic.runtime.migration] Running upgrade a2c32e6c7729 -> 1cdc775ca98f, Drop `execution_date` unique constraint on DagRun.
INFO [alembic.runtime.migration] Running upgrade 1cdc775ca98f -> 522625f6d606, Add tables for backfill.
INFO [alembic.runtime.migration] Running upgrade 522625f6d606 -> 16cbcb1c8c36, Remove redundant index.
INFO [alembic.runtime.migration] Running upgrade 16cbcb1c8c36 -> 44eabb1904b4, Update dag_run_note.user_id and task_instance_note.user_id columns to String.
INFO [alembic.runtime.migration] Running upgrade 44eabb1904b4 -> 0d9e73a75ee4, Add name and group fields to DatasetModel.
INFO [alembic.runtime.migration] Running upgrade 0d9e73a75ee4 -> c3389cd7793f, Add backfill to dag run model.
INFO [alembic.runtime.migration] Running upgrade c3389cd7793f -> 5a5d66100783, Add AssetActive to track orphaning instead of a flag.
INFO [alembic.runtime.migration] Running upgrade 5a5d66100783 -> fb2d4922cd79, Tweak AssetAliasModel to match AssetModel after AIP-76.
INFO [alembic.runtime.migration] Running upgrade fb2d4922cd79 -> 3a8972ecb8f9, Add exception_reason and logical_date to BackfillDagRun.
INFO [alembic.runtime.migration] Running upgrade 3a8972ecb8f9 -> 05234396c6fc, Rename dataset as asset.
/usr/local/lib/python3.9/contextlib.py:126 SAWarning: WARNING: SQL-parsed foreign key constraint '('alias_id', 'asset_alias', 'id')' could not be located in PRAGMA foreign_keys for table asset_alias_asset
/usr/local/lib/python3.9/contextlib.py:126 SAWarning: WARNING: SQL-parsed foreign key constraint '('dataset_id', 'asset', 'id')' could not be located in PRAGMA foreign_keys for table asset_alias_asset
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/alembic/operations/batch.py", line 690, in drop_constraint
const = self.named_constraints.pop(const.name)
KeyError: 'dataset_alias_dataset_alias_id_fkey'
There was a problem hiding this comment.
Warnings:
/usr/local/lib/python3.9/contextlib.py:126 SAWarning: WARNING: SQL-parsed foreign key constraint '('alias_id', 'asset_alias', 'id')' could not be located in PRAGMA foreign_keys for table asset_alias_asset
/usr/local/lib/python3.9/contextlib.py:126 SAWarning: WARNING: SQL-parsed foreign key constraint '('dataset_id', 'asset', 'id')' could not be located in PRAGMA foreign_keys for table asset_alias_asset
/usr/local/lib/python3.9/contextlib.py:126 SAWarning: WARNING: SQL-parsed foreign key constraint '('alias_id', 'asset_alias', 'id')' could not be located in PRAGMA foreign_keys for table asset_alias_asset_event
/usr/local/lib/python3.9/contextlib.py:126 SAWarning: WARNING: SQL-parsed foreign key constraint '('event_id', 'asset_event', 'id')' could not be located in PRAGMA foreign_keys for table asset_alias_asset_event
4ac20ad
to
4538776
Compare
ephraimbuddy
added
the
full tests needed
ephraimbuddy
force-pushed
the
investigate-migrations
branch
from
4538776
to
5f5adbc
Compare
kaxil
reviewed
Nov 22, 2024
airflow/migrations/versions/0032_3_0_0_drop_execution_date_unique.py
Outdated
Show resolved
Hide resolved
| local_cols=["asset_id"], | ||
| remote_cols=["id"], | ||
| ondelete="CASCADE", | ||
| ) |
There was a problem hiding this comment.
Yeah last I had tried I ran into:
Performing upgrade to the metadata database sqlite:////root/airflow/sqlite/airflow.db
[2024-10-27T22:43:35.430+0000] {migration.py:215} INFO - Context impl SQLiteImpl.
[2024-10-27T22:43:35.431+0000] {migration.py:218} INFO - Will assume non-transactional DDL.
[2024-10-27T22:43:35.432+0000] {migration.py:215} INFO - Context impl SQLiteImpl.
[2024-10-27T22:43:35.433+0000] {migration.py:218} INFO - Will assume non-transactional DDL.
[2024-10-27T22:43:35.433+0000] {db.py:1171} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade 22ed7efa9da2 -> 044f740568ec, Drop ab_user.id foreign key.
INFO [alembic.runtime.migration] Running upgrade 044f740568ec -> d0f1c55954fa, Remove SubDAGs: `is_subdag` & `root_dag_id` columns from DAG table.
INFO [alembic.runtime.migration] Running upgrade d0f1c55954fa -> 0bfc26bc256e, Rename DagModel schedule_interval to timetable_summary.
INFO [alembic.runtime.migration] Running upgrade 0bfc26bc256e -> a2c32e6c7729, Add triggered_by field to DagRun.
INFO [alembic.runtime.migration] Running upgrade a2c32e6c7729 -> 1cdc775ca98f, Drop `execution_date` unique constraint on DagRun.
INFO [alembic.runtime.migration] Running upgrade 1cdc775ca98f -> 522625f6d606, Add tables for backfill.
INFO [alembic.runtime.migration] Running upgrade 522625f6d606 -> 16cbcb1c8c36, Remove redundant index.
INFO [alembic.runtime.migration] Running upgrade 16cbcb1c8c36 -> 44eabb1904b4, Update dag_run_note.user_id and task_instance_note.user_id columns to String.
INFO [alembic.runtime.migration] Running upgrade 44eabb1904b4 -> 0d9e73a75ee4, Add name and group fields to DatasetModel.
INFO [alembic.runtime.migration] Running upgrade 0d9e73a75ee4 -> c3389cd7793f, Add backfill to dag run model.
INFO [alembic.runtime.migration] Running upgrade c3389cd7793f -> 5a5d66100783, Add AssetActive to track orphaning instead of a flag.
INFO [alembic.runtime.migration] Running upgrade 5a5d66100783 -> fb2d4922cd79, Tweak AssetAliasModel to match AssetModel after AIP-76.
INFO [alembic.runtime.migration] Running upgrade fb2d4922cd79 -> 3a8972ecb8f9, Add exception_reason and logical_date to BackfillDagRun.
INFO [alembic.runtime.migration] Running upgrade 3a8972ecb8f9 -> 05234396c6fc, Rename dataset as asset.
/usr/local/lib/python3.9/contextlib.py:126 SAWarning: WARNING: SQL-parsed foreign key constraint '('alias_id', 'asset_alias', 'id')' could not be located in PRAGMA foreign_keys for table asset_alias_asset
/usr/local/lib/python3.9/contextlib.py:126 SAWarning: WARNING: SQL-parsed foreign key constraint '('dataset_id', 'asset', 'id')' could not be located in PRAGMA foreign_keys for table asset_alias_asset
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/alembic/operations/batch.py", line 690, in drop_constraint
const = self.named_constraints.pop(const.name)
KeyError: 'dataset_alias_dataset_alias_id_fkey'
| local_cols=["asset_id"], | ||
| remote_cols=["id"], | ||
| ondelete="CASCADE", | ||
| ) |
There was a problem hiding this comment.
Warnings:
/usr/local/lib/python3.9/contextlib.py:126 SAWarning: WARNING: SQL-parsed foreign key constraint '('alias_id', 'asset_alias', 'id')' could not be located in PRAGMA foreign_keys for table asset_alias_asset
/usr/local/lib/python3.9/contextlib.py:126 SAWarning: WARNING: SQL-parsed foreign key constraint '('dataset_id', 'asset', 'id')' could not be located in PRAGMA foreign_keys for table asset_alias_asset
/usr/local/lib/python3.9/contextlib.py:126 SAWarning: WARNING: SQL-parsed foreign key constraint '('alias_id', 'asset_alias', 'id')' could not be located in PRAGMA foreign_keys for table asset_alias_asset_event
/usr/local/lib/python3.9/contextlib.py:126 SAWarning: WARNING: SQL-parsed foreign key constraint '('event_id', 'asset_event', 'id')' could not be located in PRAGMA foreign_keys for table asset_alias_asset_event
airflow/migrations/versions/0041_3_0_0_rename_dataset_as_asset.py
Outdated
Show resolved
Hide resolved
airflow/migrations/versions/0050_3_0_0_remove_pickled_data_from_xcom_table.py
Outdated
Show resolved
Hide resolved
airflow/migrations/versions/0050_3_0_0_remove_pickled_data_from_xcom_table.py
Outdated
Show resolved
Hide resolved
kaxil
reviewed
Nov 22, 2024
airflow/migrations/versions/0050_3_0_0_remove_pickled_data_from_xcom_table.py
Outdated
Show resolved
Hide resolved
kaxil
reviewed
Nov 22, 2024
airflow/migrations/versions/0050_3_0_0_remove_pickled_data_from_xcom_table.py
Outdated
Show resolved
Hide resolved
There have been some inconsistences between ORM and migration files but it doesn't fail in tests. This is an attempt to fix the inconsistency and also have it fail in tests
ephraimbuddy
force-pushed
the
investigate-migrations
branch
from
5f5adbc
to
47128e9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There have been some inconsistences between ORM and migration files but it doesn't fail in tests. This is an attempt to fix the inconsistency and also have it fail in tests